This PR optimizes downloads through the TransferManager by removing the upfront HEAD request on an opt-in basis. Previously, every download issued a HEAD request to determine object size before starting the GET. This change eliminates that extra round-trip by extracting metadata from the first GET response instead, when the caller has configured response_checksum_validation='when_required' on the client. The default path still uses HEAD so that full-object checksum validation continues to work.

For small files, the download time is dominated by request latency, so eliminating one of the two requests results in a ~50% download time reduction. For large files, the effect is less noticeable because the download time is dominated by the transfer itself and there are multiple chunks. In both cases, we save the cost of the HEAD request.

What Changed

• No-HEAD download path added, gated on response_checksum_validation='when_required': Downloads on that path start immediately with a ranged GET request for the first chunk. The default path is unchanged.
• Dynamic size detection (no-HEAD path): Extract object size and ETag from the first GET response headers (ContentRange or ContentLength).
• Dynamic chunk scheduling (no-HEAD path): After the first chunk completes, schedule additional chunks only if the object is larger than the chunk size.
• ETag consistency for all chunks: When an ETag is available (either pre-provided via a subscriber, e.g. from a prior HEAD request by the AWS CLI, or extracted from the first GET response), all subsequent ranged GET requests include an IfMatch header with that ETag. This ensures S3 rejects the request if the object changes mid-download. The first GET request also includes IfMatch if an ETag was pre-provided before the download started.
• Zero-byte object handling (no-HEAD path): The first ranged GET on a 0-byte object returns InvalidRange. The new path handles this by returning an empty response instead of retrying with a non-ranged GET.

Testing

Unit, functional, and integ tests pass. I also added a new script to benchmark downloading many small files (scripts/performance/time-batch-download.py). For 1000 × 1 KB files on my laptop, total download duration dropped ~41% (from 15.0s to 8.9s) on the opt-in path.

Backward Compatibility

External API unchanged. All download methods have the same signatures. The default path (HEAD + size-branching GET) is preserved, so no behavior change for callers who don't set response_checksum_validation='when_required'.

Flow Diagrams

Default path (unchanged)

flowchart TD
    A[HEAD Request] --> C{Size < 8MB?}
    C -->|Yes| D[GET Request]
    C -->|No| E[Multiple GET Requests]
    D --> F[Complete]
    E --> F

No-HEAD path (opt-in via response_checksum_validation="when_required")

flowchart TD
    A[GET First Chunk] --> B{Size < 8MB?}
    B -->|Yes| C[Complete]
    B -->|No| D[GET Remaining Chunks]
    D --> C

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.